Descriptive checks

Here we provide a descriptive overview of the syllable identifications relative to target (table 1). In the current data the number of syllables identified by EasyAlign perfectly matched the targeted number of syllables, i.e., in 100% of the trials there were 0 differences in the number of syllable detected versus target. Note however that we will manually check the EasyAllign syllable detections for the final dataset.

Table 1. Percentage of syllable detection mismatches (#tab:table01)
A summary table of percentage of differences between syllables
syllable differences percentage
0 100.00

Table 2 provides the percentages of different type of L2 stress placement matches and mismatches.

Table 2. Percentage of correct L2 placements (#tab:unnamed-chunk-1)
A summary table of percentage of stress match/mismatch types
stress mis/match type stress difference percentage
L2 correct same 33.93
L2 incorrect & L1 match same 0.00
L2 incorrect & L1 mismatch same 16.07
L2 correct difference 44.05
L2 incorrect & L1 match difference 0.00
L2 incorrect & L1 mismatch difference 5.95

Main Confirmatory Analysis

Gesture vs. no gesture

For the first analysis we simply assess whether the absolute difference in stress timing relative to the target stress time is different for the gesture or the no gesture condition.

D$accuracy <- abs(D$stressed_mistimingL2L1) #absolute deviation from stress from L2

#basemodel predicting the overall mean accuracy
model0 <- lme(accuracy~1, data = D, random = list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)

#alternative model with gesture versus no gesture as predictor
model1 <- lme(accuracy~condition, data = D, random =  list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)
anova(model0, model1) #test difference basemodel versus model 1
##        Model df      AIC      BIC    logLik   Test  L.Ratio p-value
## model0     1  4 4069.241 4084.509 -2030.620                        
## model1     2  5 4066.132 4085.217 -2028.066 1 vs 2 5.109432  0.0238
sum1 <- summary(model1) 

Click here for model 1 summary

```r
sum1
```

```
## Linear mixed-effects model fit by maximum likelihood
##  Data: D 
##        AIC      BIC    logLik
##   4066.132 4085.217 -2028.066
## 
## Random effects:
##  Formula: ~1 | ppn
##         (Intercept)
## StdDev:  0.01107632
## 
##  Formula: ~1 | target %in% ppn
##         (Intercept) Residual
## StdDev:     58.9668 85.65074
## 
## Fixed effects: accuracy ~ condition 
##                      Value Std.Error  DF   t-value p-value
## (Intercept)       62.19643  8.046698 167  7.729435  0.0000
## conditiongesture -21.28571  9.373199 167 -2.270912  0.0244
##  Correlation: 
##                  (Intr)
## conditiongesture -0.582
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.7144678 -0.4332536 -0.1847360 -0.1847359  4.1253449 
## 
## Number of Observations: 336
## Number of Groups: 
##             ppn target %in% ppn 
##               2             168
```

It is of course possible that there is a contextual effect of gesture, depending on whether there is a stress difference or a presence of an accent (see Figure 2)

Figure 2.

We will further assess this in a complex model we expand our analysis with the relevant stimuli conditions, as well as their interactions with the gesture condition. If the interactions are statistically reliable we will perform a post-hoc comparisons with R-package “lsmeans” with a bonferroni correction.

#alternative model with gesture versus no gesture as predictor
model2 <- lme(accuracy~condition+stress+accent, data = D, random =  list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)
model3 <- lme(accuracy~condition+stress+accent+
                condition*stress+
                condition*accent, 
              data = D, random =  list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)


anova(model1, model2, model3) #test difference basemodel versus model 1
##        Model df      AIC      BIC    logLik   Test  L.Ratio p-value
## model1     1  5 4066.132 4085.217 -2028.066                        
## model2     2  7 4067.404 4094.124 -2026.702 1 vs 2 2.727168  0.2557
## model3     3  9 4070.265 4104.619 -2026.132 2 vs 3 1.139483  0.5657
#summary model 3 post hoc
sum3 <- summary(model3)
posthoc3a <- lsmeans(model3, list(pairwise ~ condition|stress),  adjust="bonferroni")
posthoc3b <- lsmeans(model3, list(pairwise ~ condition|accent),  adjust="bonferroni")

Click here for model 3 summary

```r
sum3
```

```
## Linear mixed-effects model fit by maximum likelihood
##  Data: D 
##        AIC      BIC    logLik
##   4070.265 4104.619 -2026.132
## 
## Random effects:
##  Formula: ~1 | ppn
##         (Intercept)
## StdDev: 0.009244317
## 
##  Formula: ~1 | target %in% ppn
##         (Intercept) Residual
## StdDev:    58.19645 85.36076
## 
## Fixed effects: accuracy ~ condition + stress + accent + condition * stress +      condition * accent 
##                                           Value Std.Error  DF   t-value p-value
## (Intercept)                            62.29167  13.93053 165  4.471592  0.0000
## conditiongesture                       -9.94048  16.27766 165 -0.610682  0.5423
## stressdifference                      -18.29762  16.08559 164 -1.137516  0.2570
## accentaccent present                   18.10714  16.08559 164  1.125675  0.2619
## conditiongesture:stressdifference      -3.00000  18.79582 165 -0.159610  0.8734
## conditiongesture:accentaccent present -19.69048  18.79582 165 -1.047599  0.2964
##  Correlation: 
##                                       (Intr) cndtng strssd accntp cndtn:
## conditiongesture                      -0.584                            
## stressdifference                      -0.577  0.337                     
## accentaccent present                  -0.577  0.337  0.000              
## conditiongesture:stressdifference      0.337 -0.577 -0.584  0.000       
## conditiongesture:accentaccent present  0.337 -0.577  0.000 -0.584  0.000
## 
## Standardized Within-Group Residuals:
##         Min          Q1         Med          Q3         Max 
## -1.60812794 -0.46910650 -0.23332169 -0.08683539  4.28520679 
## 
## Number of Observations: 336
## Number of Groups: 
##             ppn target %in% ppn 
##               2             168
```

Click here for posth-hoc3a output

```r
posthoc3a
```

```
## $`lsmeans of condition | stress`
## stress = same:
##  condition lsmean   SE df lower.CL upper.CL
##  nogesture   71.3 11.4  1    -73.2      216
##  gesture     51.6 11.4  1    -93.0      196
## 
## stress = difference:
##  condition lsmean   SE df lower.CL upper.CL
##  nogesture   53.0 11.4  1    -91.5      198
##  gesture     30.3 11.4  1   -114.3      175
## 
## Results are averaged over the levels of: accent 
## Degrees-of-freedom method: containment 
## Confidence level used: 0.95 
## 
## $`pairwise differences of condition | stress`
## stress = same:
##  2                   estimate   SE  df t.ratio p.value
##  nogesture - gesture     19.8 13.3 165 1.489   0.1385 
## 
## stress = difference:
##  2                   estimate   SE  df t.ratio p.value
##  nogesture - gesture     22.8 13.3 165 1.714   0.0883 
## 
## Results are averaged over the levels of: accent 
## Degrees-of-freedom method: containment
```

Click here for posth-hoc3b output

```r
posthoc3a
```

```
## $`lsmeans of condition | stress`
## stress = same:
##  condition lsmean   SE df lower.CL upper.CL
##  nogesture   71.3 11.4  1    -73.2      216
##  gesture     51.6 11.4  1    -93.0      196
## 
## stress = difference:
##  condition lsmean   SE df lower.CL upper.CL
##  nogesture   53.0 11.4  1    -91.5      198
##  gesture     30.3 11.4  1   -114.3      175
## 
## Results are averaged over the levels of: accent 
## Degrees-of-freedom method: containment 
## Confidence level used: 0.95 
## 
## $`pairwise differences of condition | stress`
## stress = same:
##  2                   estimate   SE  df t.ratio p.value
##  nogesture - gesture     19.8 13.3 165 1.489   0.1385 
## 
## stress = difference:
##  2                   estimate   SE  df t.ratio p.value
##  nogesture - gesture     22.8 13.3 165 1.714   0.0883 
## 
## Results are averaged over the levels of: accent 
## Degrees-of-freedom method: containment
```

Prosodic modulation of gesture

We perform a mixed linear regression with normalized acoustic markers as DV, and acoustic marker (peak F0, peak envelope, and duration) x condition as independent variable.

Figure x.

Dlong <- gather(D, "marker", "acoust_out", 13:15)

#alternative model with gesture versus no gesture as predictor
model0 <- lme(acoust_out~1, data = Dlong, random =  list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)
model1 <- lme(acoust_out~marker*condition, data = Dlong, random =  list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)
anova(model0, model1) #test difference basemodel versus model 1
##        Model df      AIC      BIC    logLik   Test  L.Ratio p-value
## model0     1  4 1951.533 1971.196 -971.7666                        
## model1     2  9 1534.941 1579.182 -758.4703 1 vs 2 426.5926  <.0001
#summary model 3 post hoc
sum1 <- summary(model1)
posthocsum1 <- lsmeans(model1, list(pairwise ~ condition|marker),  adjust="bonferroni")

Click here for model 1 summary

```r
sum1
```

```
## Linear mixed-effects model fit by maximum likelihood
##  Data: Dlong 
##        AIC      BIC    logLik
##   1534.941 1579.182 -758.4703
## 
## Random effects:
##  Formula: ~1 | ppn
##          (Intercept)
## StdDev: 1.375216e-05
## 
##  Formula: ~1 | target %in% ppn
##         (Intercept) Residual
## StdDev:  0.08539542 0.506822
## 
## Fixed effects: acoust_out ~ marker * condition 
##                                     Value  Std.Error  DF   t-value p-value
## (Intercept)                     1.5512255 0.03977187 835  39.00308  0.0000
## markerpeakF0z                  -0.5074599 0.05546413 835  -9.14933  0.0000
## markersDURz                    -0.7038124 0.05546413 835 -12.68951  0.0000
## conditiongesture                0.1850414 0.05546413 835   3.33624  0.0009
## markerpeakF0z:conditiongesture -0.1713849 0.07843812 835  -2.18497  0.0292
## markersDURz:conditiongesture   -0.3458323 0.07843812 835  -4.40898  0.0000
##  Correlation: 
##                                (Intr) mrkrF0 mrkDUR cndtng mrkF0:
## markerpeakF0z                  -0.697                            
## markersDURz                    -0.697  0.500                     
## conditiongesture               -0.697  0.500  0.500              
## markerpeakF0z:conditiongesture  0.493 -0.707 -0.354 -0.707       
## markersDURz:conditiongesture    0.493 -0.354 -0.707 -0.707  0.500
## 
## Standardized Within-Group Residuals:
##         Min          Q1         Med          Q3         Max 
## -4.32238038 -0.52000609  0.00753372  0.54513919  4.64779877 
## 
## Number of Observations: 1008
## Number of Groups: 
##             ppn target %in% ppn 
##               2             168
```

Click here for posthoc model 1

```r
posthoc3a
```

```
## $`lsmeans of condition | stress`
## stress = same:
##  condition lsmean   SE df lower.CL upper.CL
##  nogesture   71.3 11.4  1    -73.2      216
##  gesture     51.6 11.4  1    -93.0      196
## 
## stress = difference:
##  condition lsmean   SE df lower.CL upper.CL
##  nogesture   53.0 11.4  1    -91.5      198
##  gesture     30.3 11.4  1   -114.3      175
## 
## Results are averaged over the levels of: accent 
## Degrees-of-freedom method: containment 
## Confidence level used: 0.95 
## 
## $`pairwise differences of condition | stress`
## stress = same:
##  2                   estimate   SE  df t.ratio p.value
##  nogesture - gesture     19.8 13.3 165 1.489   0.1385 
## 
## stress = difference:
##  2                   estimate   SE  df t.ratio p.value
##  nogesture - gesture     22.8 13.3 165 1.714   0.0883 
## 
## Results are averaged over the levels of: accent 
## Degrees-of-freedom method: containment
```

Gesture-speech asynchrony as a function of trial conditions

In the previous analyses we know whether speech prosody performance increases or decreases as a function of gesture, stress difference, and accentedness. A further question is whether gesture-speech synchrony is affected by stress difference and accentedness.

Figure 3.

It seems that synchrony between gesture and speech is best in the accented and stress match condition.

```
##        Model df      AIC      BIC    logLik   Test   L.Ratio p-value
## model0     1  4 2044.734 2057.230 -1018.367                         
## model1     2  6 2047.489 2066.233 -1017.745 1 vs 2 1.2451018  0.5366
## model2     3  7 2049.444 2071.312 -1017.722 2 vs 3 0.0448689  0.8322
```

Gesture-speech asynchrony and the directionality of error

Now we should know whether gesture-speech synchrony can be affected by trial condition that may complicate correct stress placement. If indeed gesture-speech synchrony is affected, we can wonder about how gesture and speech diverge when they are more asynchronous. Firstly assess whether gestures

#basemodel predicting the overall mean accuracy
model0 <- lme(asynchrony_L2L1~1, data = subD, random = list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)

#alternative model with gesture versus no gesture as predictor
model1 <- lme(asynchrony_L2L1~stress*correct, data = subD, random =  list(~1|ppn, ~1|target), method = "ML", na.action = na.exclude)
anova(model0, model1) #test difference basemodel versus model 1
##        Model df      AIC      BIC    logLik   Test  L.Ratio p-value
## model0     1  4 2133.622 2146.118 -1062.811                        
## model1     2  7 2126.648 2148.515 -1056.324 1 vs 2 12.97427  0.0047
summary(model1) 
## Linear mixed-effects model fit by maximum likelihood
##  Data: subD 
##        AIC      BIC    logLik
##   2126.648 2148.515 -1056.324
## 
## Random effects:
##  Formula: ~1 | ppn
##         (Intercept)
## StdDev: 0.007961735
## 
##  Formula: ~1 | target %in% ppn
##         (Intercept) Residual
## StdDev:    130.1349 2.115984
## 
## Fixed effects: asynchrony_L2L1 ~ stress * correct 
##                                                         Value Std.Error  DF
## (Intercept)                                          14.14516  16.72969 163
## stressdifference                                     30.74958  22.54347 163
## correctL2 incorrect & L1 mismatch                    49.44575  32.69010 163
## stressdifference:correctL2 incorrect & L1 mismatch -210.84048  58.87324 163
##                                                      t-value p-value
## (Intercept)                                         0.845512  0.3991
## stressdifference                                    1.364013  0.1744
## correctL2 incorrect & L1 mismatch                   1.512560  0.1323
## stressdifference:correctL2 incorrect & L1 mismatch -3.581262  0.0005
##  Correlation: 
##                                                    (Intr) strssd cLi&Lm
## stressdifference                                   -0.742              
## correctL2 incorrect & L1 mismatch                  -0.512  0.380       
## stressdifference:correctL2 incorrect & L1 mismatch  0.284 -0.383 -0.555
## 
## Standardized Within-Group Residuals:
##          Min           Q1          Med           Q3          Max 
## -0.063643564 -0.007597103 -0.002391494  0.008405780  0.050266435 
## 
## Number of Observations: 168
## Number of Groups: 
##             ppn target %in% ppn 
##               2             168

Power analysis

https://link.springer.com/article/10.3758/s13428-021-01546-0